Processing for a lo/l1b/goodtimes product#3117
Processing for a lo/l1b/goodtimes product#3117vineetbansal wants to merge 4 commits intoIMAP-Science-Operations-Center:devfrom
Conversation
vineetbansal
force-pushed
the
vb/lo_pipeline_merge
branch
from
af822aa to
378b5bd
Compare
vineetbansal
changed the title
vineetbansal
changed the title
tmplummer
left a comment
tmplummer
left a comment
There was a problem hiding this comment.
Just one major concern with this implementation. I left some other relevant comments too.
| start_et_hk = ( | ||
| hk_epoch_ets[0] + timedelta(hours=PIVOT_HK_HOUR_RANGE[0]).total_seconds() | ||
| ) | ||
| end_et_hk = ( | ||
| hk_epoch_ets[0] + timedelta(hours=PIVOT_HK_HOUR_RANGE[1]).total_seconds() | ||
| ) |
There was a problem hiding this comment.
Fragile code. Nominal pointings are close to 24 hours, but not all pointings are that long. For example around the TCM on April 7, the repoint table has the following:
513165615,0,513165816,0,2026-04-06 10:00:00.748931,2026-04-06 10:03:21.748756,208
513252017,0,513252218,0,2026-04-07 10:00:02.673645,2026-04-07 10:03:23.673469,209
513287744,0,513287946,0,2026-04-07 19:55:29.642514,2026-04-07 19:58:51.642338,210
513288856,0,513289642,0,2026-04-07 20:14:01.641545,2026-04-07 20:27:07.640860,211
513338414,0,513338796,0,2026-04-08 09:59:59.598459,2026-04-08 10:06:21.598127,212
Pointing 209 is ~9:52 in duration, pointing 210 is ~0:16 in duration, etc.
There was a problem hiding this comment.
I'll bring this up with @nschwadron tomorrow..
There was a problem hiding this comment.
Good point. The current implementation assumes a nominal long pointing and uses a fixed 3–15 hour window from the beginning of the HK file to estimate the pivot angle. That approach may not be robust for shortened pointings, such as those around TCM activities. Defining the HK selection window dynamically based on the actual pointing duration sounds like a better approach. I would recommend applying only a small edge trim to avoid transition periods.
| # Per-day overrides for the anti-RAM background rate. Keyed by (year, day-of-year) | ||
| BG_RATE_ANTI_RAM_OVERRIDES: dict[tuple[int, int], float] = { | ||
| (2026, 62): 0.0014, | ||
| (2026, 64): 0.0, | ||
| (2026, 65): 0.0, | ||
| (2026, 91): 0.03, | ||
| } |
There was a problem hiding this comment.
Hard coded override days 😬. Two major concerns with this:
- This required code updates and build/deploy of the Lo container each time an update to these is desired.
- Per the PI, the pipeline should be fully automated with no manual intervention to tweak algorithm parameters.
There was a problem hiding this comment.
Let me point @nschwadron to this comment - we could put these in a csv but that won't address your concerns. I'm not sure if these overrides are derivable from someplace..
There was a problem hiding this comment.
One possible approach could be to move these exception values into an ancillary file rather than hard-coding them in the algorithm.
| end = met[index - 1] | ||
| logger.debug(f" Closing interval due to time gap: {begin} -> {end}") | ||
| # Pre-compute expected exposure times [s] for the averaging and summing windows. | ||
| exposure = c.HISTOGRAM_CYCLE_EPOCHS * c.N_CYCLE_AVE * c.EXPOSURE_FACTOR |
There was a problem hiding this comment.
How will hard coded exposure times work on off-nominal pointings?
There was a problem hiding this comment.
The exposure values here are tied to the histogram cadence rather than the total pointing duration. The algorithm processes however many histogram cycles are present in the file, while each cycle still corresponds to a fixed 420-second interval. In our current runs, we use a 7-cycle averaging window (~50 minutes) that advances one cycle at a time through the file. So shorter or off-nominal pointings would mainly result in fewer total cycles being analyzed, rather than changing the effective exposure per rate estimate.
|
Does this work have an associated issue? It would be good to keep with best practices of capturing all work in a issue and linking PRs using the "Closes: #nnnn" |
|
Opened issue #3135 for this PR specifically. |
3 participants
Change Summary
Changed the
goodtimesLo/l1b product algorithm. Both the algorithm as well as the structure of the output datasets have changed. I've run the algorithm by the instrument team. Since no other products depend on these, this should be okay.Overview
File changes
l1b_bgrates_and_goodtimesfunction inimap_processing/lo/l1b/lo_l1b.pychanged to adhere to new algorithm from IMAP-Lo team (not yet officially documented). Tests related to this functionl1b_bgrates_and_goodtimeshave been tweaked.Testing
Modified tests to test some edge cases for the new algorithm. Not all edge cases have been tested, but I'll add more in a subsequent commit.